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ABSTBACT 

A coaputer* based test adainistration and aonitoring 
systea developed at the University of Delaware is described. Using 
six terainals connected to a Burroughs B*6VC0 coaputer, the systea 
provides nuaerous capabilities for interactive testing and evaluation 
of students* course learning, while also providing statistical data 
to facu'^ty in a variety of foraats* Student attitudes toward the 
systea have been aixed, although there is a generally positive 
reaction to the self-pacing features of the systea. Faculty have used 
the data analysis features extensively and have been coapliaentary of 
thea« So^e probleas and opportunities for further developaent of the 
svstea are also presented* (D6C) 
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Many college and public school teachers have, 
%#ithin the last few years, found that the tradi- 
tional lecture and test method of instruction does 
not Meet the needs of their students for adequate 
aastcry of course materials and interest in course 
content. This has led to a seirch for new instruc- 
tional methods and has resulted in the implemen- 
tation of open classrooms » independent study, 
Modularized instruction, performance or competency 
based instruction, and other non-traditional 
Methods. A necessary ingredient for the success 
of several of these methods is an effective test- 
ing program which incorporates the development 
and administration of test items and the analysis 
of results. Several people across the country 
have recently started to use the computer for help 
In the testing process because of its storage 
capacity, ability co manipulate large data bases, 
and analytical capabilities. Most test-related 
uses of the computer have been oriented toward 
test scoring and the development and evaluation of 
large pools of items. For example, tests might be 
printed by the computer and copied for the stu- 
dents who respond using mark sense f >rmfi. These 
forMS are fed into the computer with item identi- 
fication information and the resulting data may 
bs stored and used for later analyses. An excel- 
lent review of several computer assisted test 
construction projects is providcni in the March, 
1973 issue of Educational Tectoology . 

Origins of the Computer 
Assisted Testing Project 

At the the University of Delaware, several 
faculty in the College of Education have devel- 
oped courses that are partially or completely 
designed to permit independent stud/. They were 
especially concerned with ways of permitting 
students to demonstrate mastery of each curric- 
ulum module or unit. It was believed that the 
leaxning process could be improved by permitting 
a student more than one attempt it a criterion 
test# but this would require several different 
copies of a test for each unit. To solve this 
problon, the faculty turned to the computer. 

In 1969 a computer program was developed by 
Teresa Green to generate randomly parallel tests 
with s specified number of questions from a large 
pool of test bluest ions. Test questions were ran- 
dOMly selected so that any item might appear on 
any copy of the test- Lut the same item would not 



appear more than once on a particular test. A 
blank was provided for the student to enter his 
response to each question, and a separate strip of 
the output %*as reserved for the answers. This 
answer strip could be separated from the printout 
containing the questions befc -e the' test was given 
to the student. An instructor could then compo 2 
the strip with the student responses and easily 
grade the test. The randomization of the questions 
made it possible for a student to take a test t%#o 
or three times with a low probability of seeing a 
question repeated. Iiote that each student had 
his own copy of the test pr intent; tests did not 
need to be reproducec' in the usual way. 

The basis of this competency-based modular 
learning process was carefully specified behavioral 
objectives to guide a student's study of assigncjd 
reading materials. A student's grade depended upon 
the number of points he accumulated for his work. 
The tests were criterion-referenced in that a 
pre-specified minimum score had to be achieved 
before credit could be granted for the unit. 
Clear ly# it was advantageous to permit a sttjdent 
to take More than one test on a unit to insure 
that he had reached an acceptable level of compe- 
tence* 

Based on our experience, the paper 

and pencil tests have worked reasonably i^ell, but 
they have several disadvantages. Several hundred 
teste must be generated periodically and carried 
to the testing location. Answer strips must be 
separated from the questions. Sooeone must always 
be present to distribute and score the tests and 
periodically file them. With large numbers of 
tests and students these tasks are time-consum- 
ing, arduous, and likely to result in errors. 

As with all paper and pencil tests, students 
have the opportunity to view all questions at once 
and may obtain aid from one question in answering 
another. The students, of course, nay set this ^s 
an advantage, but it may lead to large differences 
in test difficulty so that any given test may not 
provide a true indication of a student's knowle'^e. 
This is especially true if several items refer to 
one concept. For the courses that have been 
developed thus far, this problem lias not been 
especially noticeable, but the potential for 
difficulty exists. • 

tfhen test items are used repeatedly, it is 
desirable to obtain item analysis dcta for the ^ 



^Paper given at the 13th Annual Convention of the Association for Educational 
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•valiMtion of the itcu. In this way, itc«s that 
•re especially easy or difficult or which are con- 
fusing or Misleading can be detected and rewritten 
•s necessary, with paper ana pencil tests, trou- 
blesoae items can be detected through discussions 
with students, but the psychometric properties oi 
the itess cannot be determined because the random 
nature and the large number of test? makes the 
application of item analysis techniques quite 
difficult. 

The Coi^ter Assisted Testing Program 

A solution to several of the problems with 
paper an-1 pencil tests, and the next logical step 
In the testing program, was the development of an 
administration process in which the computer 
administers the test directly to the student* An 
additional reason for caking this step was the 
desire to introduce education students to com- 
puters before they encounter computer facilities 
in schools in which they will soon teach. Fuial- 
ly, this was seen as an important step toward the 
establishment of a comprehensive computer assisted 
instruction and computer managed instruction 
program which would meet a critical need within 
the College* 

• Three cathode ray tube (CRT) terminals were 
ordered and programming was initiated in December, 
1972. The first students took tests via the 
terminals in i^ril, 1973. Initially, the program 
was very simple, but extensive improvexoents have 
been made, three mere terminals have been obtain- 
ed, and, recently, upwards of 200 tests a day have 
been administered. 

The six terminals are presently connected via 
300 baud acoustic couplers to a general purpose 
A^rroughs B6700 computer that handles all the 
University's coasting. The testing center is 
open SO hours a* week and is staffed by one full 
time proctor and several pare time student proc- 
tors who provide assistance to 800 or so students 
taking the independent study courses currently on 
the system. Shortly after the stare of a term the 
students generally know what to do, thus freeing 
the proctors for additional duties such as data 
tabulation and maintenance of student files. TO 
provide a backup in the event of occasional com- 
puter down time or when the student load becomes 
too great, paper and pencil tests are available 
and are adoinistered and scorei by the proctors 
vhen required. 

In our system the students* interaction with 
the terminal was designed to be as simple as 
possible. All student responses are set up as 
formatted input; the student answers each question 
aa it is asked. By faculty reqjest, feedback is 
provided after each response, and a student con- 
tinues to respond ur.til he selects the correct 
answer. A comment option is provided that permits 
the 'student to store a narrative comment for each 
question and for the whole test. These comments 
are incorporated into the student record file for 
later review by faculty. Upor completion of the 
test, a sumomry of the perfomance is displayed. 

The program contains a number of options for 
controlling the interaction with students and for 
generating a variety of student records. Items 
can be sampled from the item pool with or without 



replacement for specific students and/ or specific 
tests. Feedback can be modified or eliminated from 
the testing process if desired. 

The data base contains identification, item 
response, response latency, and coomeiit infor- 
mation for each test administered. Several 
options for generating student records are avail- 
able. Both student-oriented and it^m-oricnted 
outputs are provided, and several parameters may 
be manipulated to select on different attributes 
in the tile. This student records system provides 
a powerful means of analyzing item responses and 
improving the item pools. 

A new addition to the program is an automatic 
records-keeping routine that automatically stores 
a student's score in a separate file. Thus, all 
scores for all tests are directly addressable for 
each student. This file is dumped periodically to 
permit students to review their progress and to in- 
sure accurar the results. Of course, at the 
end of the semester this file provides a complete 
record of each student's performance. A special 
utility routine permits the addition of scores 
from the paper and pencil tests to this student 
■cores file. 



Student and Faculty Attitudes toward CAT 



Sttident attitudes toward the use of conqniters 
for testing and instruction have been assessed 
with a 59-item questionnaire for several terms. 
Attitude information has also been gained through 
the use of course attitude surveys and through 
discussion with the students, in addition, perfor- 
mance data for one course have been analysed. 

In contrast to normal uses of atti'«.ade 
surveys, we administered the form both before and 
after testing and were able to assess changes in 
attitude. The pre-testing results ' indicate that 
students have a slightly positive attitude toward 
the use of computers for testing and instruction, 
but are concerned about the mechanical aspects 
such as the ease of operating a terminal. 

The first major use of the attitude ques- 
tionnaire was during the spring semester of 1974. 
This turned out to be a period of numerous problems 
and problem solving activities for our programming 
staff and the Computing Center staff. System 
crashes and program difficulties occurred more 
often than **ere desirable. Consequently, 
it was not surprising to find a decrease in positive 
attitude toward computer uses for testing and 
instruction. However, on the Likert-type items, 
the means of the responses were typically rather 
close to *the neutral point which, under the circum- ' 
stances, indicates that the average student will 
accept the use of computers for testing even if he 
encounters occasional computer probleais. Further- 
more, student attitudes were more positive toward 
the mechanical aspects of the operation; students 
found the terminal operation easier than antici- 
pated. 



(1) T.iis questionnaire is an extensive revision 
of one developed by Bob brown at the Pennsy- 
lvania Scate University (Brom, 1966). 
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Despite the difficulties encountered by the 
students^ an independent course evaluation con- 
^ ducted by the instructor of a psychology course 
Ifdr vhich 400 students %#crc taking testi via the 
teminels) indicated that students strongly pre- 
fezxed the testing by terminals to the nore tra- 
ditional testing process, and they thought they 
learned nore than in a "traditional" course. They 
viewed' the coaputer quizzing as a very valuable 
a^>ect of the course. AlMst all students in all 
courses gave high marks to the <^>portunity to work 
at their own pace, knowing the results after each 
question, and being able to take more than one 
test. As might be expected, there were soae 
students %*ho intensely disliked the use of term* 
inals or were afraid to use them, no matter what 
the environment. 

During the fall of 1974, the computer and 
programs worked quite well; the environment was 
. about as good as it will get. A mid-term ques- 
tionnaire was administered to 130 students, 
nineteen percent reported they were afraid to use 
the terminal or hated to use it. However, thirty 
percent enjoyed using it, 32 percent didn't mind 
using it, threa percent had no opinion, and 16 
percent preferred not to use the terminal* Most 
of the negative group seeiped to be concerned about 
seeing the whole test and being able to go back 
^nd change ansmrs. However, questionnaire 
results indicated that this problem decreased with 
increasing terminal use* 

Typically, on course evaluation question- 
naires that permit students to specify most liked 
end most disliked features of the course, the use 
of tentinals is mentioned about equally in both 
categories. This seems to be independent of the 
ability of the student, his total attitude toward 
the course, and the testing environment. 

The analysis of scores for quizzes taken by 
the im s e students on terminals and hard copy 
indicates that there is no marked difference in 
performance. Jt appears that the use of terminals 
makes little difference even though some students 
feel that it does. 

Faculty have made extensive use of the item 
analyses provided at the end of each term. 
OorrecticAS, additions, and deletions to the test 
question file may be easily made using t'.e time- 
sharing, editor. Faculty %^ have incorporated the 
CAT process into their courses have been quite 
satisfied and have continued to add units and/or 
questions to their files. 

Discussion and Summary 

The computer assisted testing model that has 
been developed has several advantages and some 
disadvantages for all participating. From the 
student's point of yiew, the advantages of this 
approach are: 

(1) The self-pacing capability allows stu- 
dents to work through the course at whatever rate 
suits them* 

(2) The provision of immediate feedback 
makes the testing activity a learning as t#ell as 
en evaluation experience. 

(3) The multipM testing opportunities 
provided for each instructional unit permit most 
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Students to achieve most of the unit objectives. 
Anxiety levels a^e typically loi#er than for tra- 
ditional tests; usually, the whole grade does not 
ride on one or two tests. 

(4) The independent study format relieves 
the student of the task of listening to lectures 
cn content he already knows or could easily learn 
on his own. 

Some of the disadvantages, from the students' 
viewpoint, relate to the instructional materials 
used and/or the lack of alternative ways of 
demonstrating achievement of unit objectives. The 
self-pacing feature allows students to procrasti- 
nate so that there are always a few students who - 
attest to take all the tests during the last 
tieek of class. • 

From the instructor's point of view, CAT 
confers at least four advantages: 

ID The level of student acnlevement is very 
high. Comprehensive pre- and post measures taken 
over several semesters have indicated that this 
method is superior to the standard lecture ap* 
proach used previously. 

(2) The role of the instr^jctor who uses the 
independent study approach changes considerably. 
The bulk of his time is now spent in developing 
and revising curricular materials (both instruc- 
tional and measurement mat** rials) and tutoring 
students experience difficulty with specific 
concepts or tecnniques. The latter activity 
provides a great deal of useful guidance for the 
curriculum development phase of his work. 

(3) If desired, supplementary activities 
which facilitate transfer of training and the 
enrichment of student leamiTig can b^ added on. 
Such activities may include lectures, field trips, 
short term internships, research projects, etc. 
The point is that the CAT model, with independent 
study, frees the instructor' of many highly routin- 
ixeft fun'Jtions associated with more conventional 
^tpvi^hes ahd allows him to engage in more inno- 
vative and profitable activities. 

(4) The independent study CAT model allows 
the instructor to turn an entire course. into an 
ongoing research laboratory in which he can study, 
in a very controlled fashion, the interrelation- 
ships which exist among CAT system variables^ 
content les, and student variables. The 
opportunity to do quality and productive instruc- 
tional research is an invaluable aspect of this 
approacn. 

^t present, three possible problem areas 
associated with CAT can be identified: 

(1) Systea reliability is of utmost isqpor- 
tance. The implementation of the sophisticated 
CAT system on a large multipurpose computer 
atteaqpting to satisfy the diverse and demanding 
user community resulted at times in an intolerable 
number of system failures. Clearly, reliability 
is a necessary condition for the success of any 
CAT implementation, fortunately, most of the 
problems seem to have been worked out and we feel 
justified in expanding our activity as additional 
resources become available. 

(2) The securi.ty of the test item bank is of 
oiNitinuing concern. The CAT approach requires a 
higher level of security but, by its very nature, 
it is a very difficult system in which to maintain 
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Mcurity. At present, security Im aaintained by 
a wmxm of user snd proj.^t numbers, through using 
the itca file in packed for«« and through using 
^»ecial keys on the CRT keyboard to permit access 
to the testing pro<yraM. 

(3) Resource allocation is a continuing 
problea. Once the success of such a system has 
been ieaonstrated, demand grows. The problem is 
further complicated by different production modes 
and research demands on the testing. The reso-* 
lution to this problem depends heavily on the 
prospects for acquiring additional resources 
and/or reordering the priorities of the academic 
units involved. 

As v^'itioned above, outstanding opportunities 
are afforded by the CAT process for conducting 
several kinds of research. A few areas for re- 
search are related to the following: 

(1) Testing time is always at a premium. In 
CMr courses, students spend a greet deal of time 
testing* Therefore, testing efficiency is of 
great concern. How can the maximw amount of 
information regarding student achievement be 
obtained in the least amount of test time? A 
nmber of researchers are currently wor*J.n<i on 
this prc^lem. Some potential answers may lie in 
sequential (or tailored) testing, in the appli- 
cation of Bayesian techniques to item selection 
and analysis, and possibly in tl-^ use of confi* 
dence measures. 

{2} the ef f3cts of variables suTh as type of 
fe ed back, spacing of testing opportunities, size 
and nature of the item banks, setting of mastery 
cut points; etc* , op. student le^xning and atti- 
tudes have yet to be systesuitically studied. 

(3) At this time, we know very little about 
the ways in which students use the CAT system. A 
catalogue of such 9;trategies, an evaluation of 
their relative effectiveness, and the determin- 
ation o^ whetiwr or not they can be successfully 
used by others is clearly appropriate and impor- - 
tant at this time. 

Computer assisted testing has unquestionably 
ei^anded the repertoire of instructional strat- 
egies available to faculty. Ho%#ever, the full 
range of the effects of such strategies on student 
learning of and attitixies toward course content 
have yet to be assessed. CAT appears to have 
great potential; a systematic exploration of its 
usefulness is now in order. 
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